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ABSTRACT 



Well-conducted, theory-guided empirical research into school 
administration is important, but will never by itself provide ultimate 
answers to questions of human action, choice, and value. This paper describes 
the theoretical framework studies conducted by the Cognitive Approaches to 
School Leadership (CASL) Project. The paper explains central constructs in 
cognitive- science theory, such as problem space, problem structure, 
expertise, schema, and implications for the study and practice of school 
administration. Two CASL studies, which compared how practicing and novice 
principals viewed and talked about their work and problems, are described. 
Finally, the paper considers two areas of continuing concern: the use of 
think-aloud protocols (in which subjects think aloud as they work on a 
presented problem) , and the difficult problem of selecting appropriate expert 
and novice subjects. (Contains 91 references.) (LMI) 



***************************************************************************** 

♦ Reproductions supplied by EDRS are the best that can be made 

♦ from the original document. 



U.S. OCPAHTMENT OF EDUCATION 

Office of Educationel Reiearch end Improvement 

EDUCATIONAL RESOURCES INFORMATION 
^ CENTER (ERIC) 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL 
HAS BEEN GRANTED BY 






This document has been reproduced as 
received from the person or organisation 
origir^ating it. 






□ Minor changes have been made to improve 
reproduction quality. 



a Points of view or opinions stated m this docu’ 
ment do not necessarily represent official 
OERI position or policy 



TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Problem processing and the principalship: 
Theoretical foundations and the expertise issue. 



Revised version of a paper presented as part of the symposium 
Clinical assessments of practical performance in school leadership: 
Findings from novice-expert studies of the elementary principalship. 
Session #41 .22 at the Annual Meeting of the 
American Educational Research Association 
New York City, April 1996 



This research reported in this paper was supported by the Social Sciences and Humanities 
Research Council of Canada under grant #410-92-0329. 



Derek J. Allison 

The University of Western Ontario 



CASL Project Paper #AERA96 1.2. revised July 1996. 



BEST COPY AVAILABLE 



2 



Theoretical foundations & the expertise issue. 

P 1 



Problem processing and the Principalship: 
Theoretical foundations and the expertise issue. 



Derek J. Allison 

More than three decades ago, Hemphill, Griffiths and Frederikson (1962) published their landmark 
account of Administrative Performance and Personality in elementary school administration. Then and now their 
detailed inquiiy^ into how 232 principals responded to a rich set of simulated administrative problems stands as the 
most ambitious, detailed, and intensive stud> of its kind available. But in addition to the sheer magnitude of the 
data collected and anah'zed, their stud> qualifies for landmark status as an exemplar of the kind of research w hich 
was promoted b>^ the New' or Theor>' Movement in educational administration. Research of this kind could be 
described as disciplined inquir>' into theor>' derived claims about the nature and processes characteristic of 
organizations such as schools, and the behaviours and other states of indi\iduals and groups who interact within 
such organizations, inquiry which makes use of and may rely extensively on attempts at measuring and analy’zing 
variation within and between sets of defined variables. Critics of the Theory Movement, notably Thom Greenfield, 
questioned the relevance of such research to the human realities of life and work within schools. In more extreme 
criticisms, disciplined inquiry into the mysteries and mechanisms of administration through attempts at measuring 
theoretically grounded or derived variables is deemed to be not just inappropriate but also epistemologically 
invalid. As such there are some who wish to proscribe empirical or — to replace an adjective with an 
epithet — empiricist research in favour of interpretive explorations of the socially constructed worlds we know' as 
schools, and the constantly contested exercise of and resistance to power which we call administration. 

A complex set of interacting motives lie behind my decision to begin this paper with the preceding 
paragraph. The purpose of the paper is to clear the conceptual ground for those that follow b>' sketching the 
relevant theory base for the unabashedly empirical research activities which are to be reported and to discuss 
sev eral central but worrisome concepts. But before jumping into the morass of detail that is needed to achiev'e these 
aims I thought it important to frame our research within the broader historical and epistemological context staked 
out b>' the first paragraph. One reason for doing so is simply to demonstrate that we are aware of that context and 
the deep and difficult questions embedded within it. Yet once we begin to immerse ourselves in the technical 
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details of the research to be presented the waters of empirical inquir> will threaten to close over our heads making 
it difficult or impossible to subsequently acknowledge alternate ways of interpreting or explaining the w orld. In 
one sense, then, the opening paragraph may serve as a set of metaphorical water-wings which will hopefully keep 
both us and you from drowTung in the empirical waters in which we must bathe. If. in addition to this, these 
water-wings help remind us all of the characteristic limitations of empirical research, they wall have served us well. 
Chief among those limitations is the narrow*, tightly focused, implicitly monochromatic view of the few' elements 
selected for stud>' which ine\ itably results. Such a view cannot hope to reflect much less fully represent the 
colourful complexities of administrative action as lived b>' participants or interpreted b>' observ ers — and it does not 
(or should not) pretend to. What it can do is identify regularities and contrasts within and between otherwise 
inaccessible states and processes w'hich our best theories portray as underlying and constraining social action in 
general and administrative practice in particular. When empirical inquiiy succeeds in this endeavour — and it may 
fail as often as not — then we are rewarded with potentially important nuggets of information w hich ma> help us 
better understand the limited area of inqiury addressed b>' the research responsible, and which may therefore 
contribute to building or modifying broader accounts of phenomena and processes of interest. In either case the 
findings which result and, more importantly, the manner(s) in which the>^ come to be interpreted and understood, 
will help us or others find what appropriately placed arbitrators will eventually decide are better ways of doing 
important things. That is all that may reasonably be expected of empirical research. One of the unfortunate faults 
associated with the advent of the Theor>' Movement was a misplaced belief that properly conducted empirical 
research would produce something akin to a universal technolog>’ for the effective and efficient practice of 
administration. Such an idea grossly misrepresents the nature of the social world, the task of social science in 
inquiring into that world, the role of theoiy in building tentative explanations of aspects of that world, and the 
powers of empirical research in gathering potentially relevant information. This is not to say or suggest that we 
should abandon theor>' guided research into educational organizations and their administration in favour of the 
many flavours of interpretive inquir>’ now' in vogue. (Juite the contrar>'. In our \iew' well conducted, theor>' guided 
empirical research into schools and their administration is more important than ever, provided that we recognize 
and accept that it will never. b>* itself, provide ultimate answers to the enduring questions of human action, choice 
and value. Nor. of course, will any of the trench* interpretive approaches. All that we may reasonably expect is a 
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stead>' and hopefully not-too-slow accumulation of reasonably reliable empirically grounded information 
augmented b>^ insights and appreciations from interpretive inquiiy^ which we will try* to interweave into 
incrementally better accounts — theories if you like — of phenomena and processes of interest and importance. This 
is the broad end to which we sought to contribute in the research to be reported. 

Hemphilf Griffiths and Frederikson (1962) justified their area of inquiry^ as follows: 

The administrative performance of the elementar>' school principal was chosen because of the 
high degree of interest in the problems associated with the principalship. Boards of Education, 
superintendents of schools, and the general public are demanding improved methods of 
educating and selecting school principals and of improving the administration of schools, (p.l) 

And so it goes. Our reasons, along with those given by^ many others who have studied school principals in recent 

decades, were much the same, heightened b>^ the steadily increasing importance of the principalship as the broad 

trend tow'ard devolving increased responsibility' to schools proceeds apace. That w'e and others should choose to 

focus for our work on a common topic is as it should be. The elementary principalship is the most numerous post 

in the profession of educational administration and is clearly a crucial one, being recognized by both theory and 

conventional wisdom as pivotally important in creating and sustaining good schools, and thus helping make or 

break the life chances of children and the future of their societies. Accumulating knowledge that will assist 

principals in enhancing their professional practice and aid aspiring principals to better prepare themselves for the 

responsibilities they seek is a project that will require a great deal of sustained empirical research, interpretive 

inquiry and synergetic theory' building. So too with the development, testing and improvement of better selection 

processes. In retrospect, the Hemphill, Griffiths and Frederikson study' was a good, useful, still relev^ant, but 

strangely neglected, start. Subsequent contributions by many other researchers have moved us along. The work 

reported in the papers prepared for this symposium* was intended to move us a little farther while making a 

contribution within the cognitive science approach to administrative work which we believe offers a promising way’ 

forward. 

Background and Context 

The research program which led to this symposium is grounded in what Leithwood and Hallinger (1993, 
p. 296) describe as the '‘loose, interdisciplinary’ alliance called cognitive science.” As discussed by Simon and 

* A list of the six papers prepared for the symposium appears at the end of the reference list. 

CASL Project Paper #AERA96 1.2, rev ised July 1996. 



BEST COPY AVAILABLE ^ 



Problem processing & the principalship 
P ^ 

Kaplan (1989) the disciplines numbered among this alliance include experimental, cognitive and social 

ps>'cholog> , computer science, neuroscience, linguistics, philosophy — especially formal logic and 

epistemology — anthropolog>^ and economics (p. 3). Simon and Kaplan identify the goal of cognitive science “as the 

stud) of intelligence and intelligent systems, with particular reference to intelligent behavior as computation” (p. 

1), or, if one prefers, information processing. Readily conceding the enduring absence of any satisfactoiy' definition 

of intelligence, Simon and Kaplan nonetheless suggest that most of us would agree 

that people are behaving intelligently when they choose courses of action that are rele\ ant to 
achie\ing their goals, when they reply coherently and appropriately to questions that are put to 
them, when they solve problems of lesser or greater difficult} , or when the>’ create or design 
something useful, beautifiil or novel, (p. 1) 

The reference to problem solving has particular significance, studies and theories of ho\^ people work on and 
through different kinds of problems having become established as a major and characteristic line of inquiiy' in 
cognitive science. Their final comment also has particular significance for it turns out that many of the interesting 
and important problems worked on by administrators imite them to design innovative solutions, at least some of 
which have their own beauty. 

Work in education has participated in the interdisciplinaiy^ alliance that is cognitive science in three main 
areas. Pioneering work on student problem-sohdng and learning by educational psychologists such as Bereiter and 
Scardamalia (1982, 1989) and Schonfeld (1985) has fuelled the development of new models of curriculum and 
instruction (Leinhardt, 1992; Murphy, 1991). Studies of the cognitive processes underlying the plans and actions 
of effective teachers as exemplified in Schulman's (1986, 1987) work and as reviewed b> Shavelson and Stern 
(1981) and Clark and Peterson (1986) have stimulated reappraisals of teacher preparation practices (Clark & 
Lampert, 1986: Floden & Klinzing, 1990) and produced models of expert teaching (Berliner, 1986; Sternberg & 
Horvath, 1995). The third area of activlK is educational administration wherein, as witnessed b>' the appearance of 
Hallinger. Leithwood and Murphy's (1993) Cognitive Perspectives on Educational Leadership and a special 
edition of Educational Administration Quarterly (August, 1 993) de\'Oted to that topic, increased attention is being 
paid to how administrators think about and through the problems that define their work. 

Problem focused approaches to educational administration are b>' no means new. The Hemphill, Griffiths 
and Frederikson stud>* incorporated a problem solving strand grounded in Hemphill’s (1958) earlier analysis, while 
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the note^^ orthy volume Problems in Educational Administration b> Strayer and his colleagues appeared as early as 
1925. Case stud>^ analysis, which is pre-eminently a problem appreciation and solving activity, has long been used 
for instructional purposes (e g. Sargent & Belisle, 1955: Hoy & Tarter, 1995), and frequent references to the 
problems of practice and the administrator’s obligation to deal with them can be found throughout the broader 
disciplinary' literature (e g. Haller & Strike, 1986: Sergiovanni, 1991). But while problem oriented and solving 
approaches to educational administration are not ne>^ , the cognitive science approach as pioneered by Leithwood 
and his colleagues constitutes an important theoretical departure from established assumptions and approaches. 

The key' disjunction lies in differences between established theories and normative models of 
administration as decision making and the applications and implications flowing from cognitive science theories of 
problem solving. It is important to stress that this distinction is rooted in differences between theories of how 
indi\iduals and organizations identity' and respond to situations, rather than an idiosy ncratic preference for 
thinking and talking of administration as decision making or problem solving. There can be no question that 
administrators make decisions or that they influence collective and individual decisions made within their 
organizations: deciding is the quintessential administrative act. But how may we best understand and inquire into 
the processes by' which administrators reach — or avoid — their decisions? The models of decision making typically 
presented in our textbooks as guides for solving administrators’ problems (e g. Hoy & Miskel, 1991, pp. 51-52 & 
Chapter 10; Hoy & Tarter, 1995: (Dwens. 1987) are plagued by' notorious limitations when it comes answering this 
question. By concentrating on ways of generating, assessing and choosing between alternatives these models 
typically focus attention on the resolution and disposition of presented problems while eliding or even ignoring 
broader information collection, processing and interpretation activities that preceded and underlie initial and final 
commitments to decision. In consequence, and as has been widely commented on by administrative philosophers 
such as Greenfield (Greenfield & Ribbins. 1993), Hodgkinson (1978) and Vickers (Open Systems Group, 1984), 
established theories of administrative decision making cannot readily accommodate the existence and influence of 
personal values on an administrator’s dispositions and actions. Moreover, as shown by Mintzberg’s (1973) 
pioneering observational study' and others that followed, established theories of decision making fail to accurately 
or even adequately reflect how administrators go about their work and actually decide matters. Cognitive science 
theories of human problem solving promise to help redress these limitations by' directing attention to how 
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administrators and others recognize, think, talk and generall} work on problems. This allows an administrator's 
values, as well as his or her professional knowledge, individual abilit> . and general reasoning to enter directly into 
theories and analyses of administrative action. As Leithwood and Hallinger (1993, p. 298-9) put the point, 
“Cognitive perspectives remind us that what administrators do depends on what the\^ think — their overt behaviours 
are a result of covert thought processes.” More comprehensively, cognitive theories focus attention on how 
administrators recognize, discover, interpret, choose, and work on problems of practice while also providing a 
framework within which to compare how' different administrators view' and work on more and less similar 
problems. 

For clarity, a further distinction needs to be made between cognitive science and other theories of problem 
solving. While the more or less rational models of decision making referred to above have provided the major 
established theoretical and normative frames for established approaches to considering problem solving in 
educational administration, contributions b>' Getzels (1979), McPherson, Crowson and Pitner (1986) and others 
have addressed administrative problem solving through various alternative frameworks. For the most part these 
frameworks were derived from Gestalt and associationist psychological theories within which successful solutions 
to problems flow from finding ways to reconceptualize situations or free otherwise blocked responses. In the 
broader literature these approaches undergird popular practical techniques such as brainstorming (Osborn, 1953), 
s> netics (Gordon, 1961) and the group problem solving principles discussed b>* Maier (1970). These approaches 
have little in common with the cognitive science derived theories of problem solving informing our work and that 
of Leithwood and his colleagues, to which we now turn. 

Theoretical Frame%wrk 

Problems are generally understood as difficult situations where the appropriate course of action is 
undecided, unclear, undesirable or all of the above. Haller and Strike’s (1986) useful definition reflects these 
elements by obser\*ing that “we have a problem when something is wrong or when something can be made better” 
(p. 1). Cognitive science approaches to problem solving readil) accommodate such definitions, but typically 
emphasize task completion under conditions of uncertainty. As Simon (1978, p. 272) put it, “a human being is 
confronted with a problem when he has accepted a task but does not know' how to carr\' it out.” As discussed in 
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greater detail elsewhere (Allison, 1996), the central constructs in cognitive science theories of how people go about 
doing this are problem space, structure and schema together with the related notion of domain expertise. 

Problem space 

Problem space is understood as the mental representation of a presented or discovered problem formed b> 
a potential solver. It is thus a conceptual analogue of how someone understands a problem. It is defined b>' the 
initial or current state, that is the uncompleted task or the situation to be made better, the goal state, this being the 
completed task, the corrected or improved situation, and the space, the “distance,” between the two. The search for 
a solution is viewed in terms of seeking a path through the problem space from the current to the goal state. This is 
attempted b> applying and manipulating “operators,” that is the rules and procedures, appropriate to the task at 
hand. A commonplace example would be writing a letter or memo. The goal state is a completed letter that w ill 
hopefully have the desired effect on the recipient. The initial state is the blank piece of paper or computer screen 
and the writer’s knowledge about the circumstances that have lead to the need for a letter as well as his or her 
technical knowledge about how to write letters. The operators that must be manipulated to cross the problem space 
and complete the task are the words and phrases needed to express what needs to be said, and the rules of grammar 
and style needed to properly, pleasingly and persuasively combine words and phrases to accomplish the goal. 

Searches for a suitable path across a problem space do not all always succeed, in which case the attempt at 
solution will be postponed subject to the acquisition of additional information, or perhaps abandoned. On occasion, 
individuals or groups may work on problems for a considerable time, searching for paths through the space, 
collecting new information to update or refresh the problem space, searching again, and so forth, before ev entually 
abandoning the search for a solution. For this and other reasons, we prefer the phrase problem processing to 
problem solving so as to recognize that not all attempts at seeking solutions to problems succeed. Moreover, 
finding an appropriate and acceptable path across a problem space — seeing how to solve a problem — will not 
necessarily lead to a quick, or indeed, any solution, for a substantial amount of work and or time may be necessary 
to modify the relevant elements of the physical or social system within which the actual situation is located. This is 
particularly so for administrators who must t>pically work through and with other people when attempting to 
resolve organizational problems, some of whom may hav e formed quite different problem spaces around alternate 
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goal states to represent their understanding of the situation, or may prefer alternate paths through a commonly 
defined space. Because of this, problem spaces formed b\* administrators to represent other than rapidly resolved 
situations will be constantly subject to refreshment and revision in the light of changing circumstances and 
unanticipated dev elopments. 

Problem structure 

The degree of structure present in a problem is represented b\’ the state of the problem space at a given 
time. Well-structured problems are those for which the processor is able to construct a accurate representation of 
current and goal states and readily identify the operators to be applied when seeking a path to the goal, as when 
writing a straight forward letter. A high degree of structure does not necessarily make the search for a solution 
path eas> or effortless, although that will be so at limes, nor does it guarantee that a solution to the problem will be 
found. It does, however, signify that the solver has a sufficiently firm grasp of the problem to start and continue 
working on it. An ill-structured problem obtains when there is uncertainfy about any or all of the three defining 
states — when the solver cannot readily identify' the current and/or goal state, and/or is unsure about the relevant 
rules or procedures for moving from one to the other. Confronted with a task for which he or she can only create 
an ill -structured problem space a potential solver may literally not know where or how to begin working on the 
problem. Some tasks appear inherently more ill-structured than others. For principals, requisitioning school 
supplies will likely appear as a more highly structured task than organizing a school trip which in turn is a more 
structured problem than revitalizing a moribund school. Yet problem structure, like beauty, is determined through 
the eye of the beholder, for what is an ill-structured problem for one person may be relatively well-structured for 
another. The difference rests on the familiarify of the task at hand and the amount of relevant knowledge and skill 
that can be brought to bear on its completion. Most of us would find the task of composing a fiigue to be an 
extremely ill-structured, indeed impossible problem, but for the mature J. S. Bach this was presumably a relatively 
straight -forward, reasonably well structured, task. So too with specialized problems in other domains. For an 
experienced school principal, organizing and superv ising a school trip will likely be a highly structured chore 
which can be attended to with little difficulty; as easily, perhaps, as most of us drive our vehicles to work in the 
morning. For a beginning principal the same task may be less well structured and thus more difficult. 
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A similar difference is considered to apply in the case of inherently more ill-structured and demanding 
problems, such as transforming a school. This is clearly a much more open, diffuse and contingent challenge than 
organizing a school trip, so much so that it qualifies as the kind of highly ill-structured task recognized as a design 
problem b> Simon (1973). Design problems characteristically require solvers to invent a unique solution, as in 
composing a fugue or transforming a particular school. The\ will also typically have an imprecise goal state — we 
have a general idea of what is required — an undesirable situation made better, a finished fugue, a transformed 
school — but the details of the final outcome are unknown when the individual or group concerned begins working 
on a design problem. Such tasks also typically involve sets of nested sub-problems, the resolution of which will 
constrain and help shape the final outcome. Working through design problems will also often take an extended 
period of time, extending over years or even decades in some cases. As shown by Reitman’s (1965) analysis of how 
a composer actually wrote a fugue and illustrated by Lighthall and Allan’s (1989) study of attempts at school 
transformation, the initial open and ill-defined problem space and goal state gradually acquire greater structure as 
progress is made, initially open constraints are closed, sequential choices made, and sub-problems are resolved in 
the pursuit of a satisfactor>’ outcome. Just as we would e?q)ect an outstanding musician such as J. S, Bach to be 
better able to set about composing a fugue than a first year music student, it seems reasonable to expect that an 
experienced and pre\ iously successful principal would be able to inject relatively greater initial structure into the 
problem of transforming a school than a recently appointed colleague. Moreover, we would expect the more 
experienced principal to be able to draw on and apply a richer set of relevant knowledge and appropriate 
procedures as she worked on transforming the problem space, and thus the school. 

Expertise 

From the argument to this point, proficient problem solving hinges on being able to draw on and apply 
knowledge and skills appropriate to the problem at hand. People who are able to do this reliably are credited with 
having acquired expertise in solving the problems associated with performing certain tasks. The limitation to 
“certain tasks” turns out to be important and interesting. Most of us can lay claim to expertise in solving the 
problems presented b\' tasks encountered in the course of everyday life. On reflection, such tasks are not inherently 
eas>’, they only appear so because of their familiarity. The every day tasks of communicating through the media of 
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oral and wTitten language, for example, or of conforming to the social mores and norms of our workplace are 
inherently complex, yet are so familiar as to be relatively unproblematic. Indeed, most of us have developed rich 
repertoires of highly automated procedures and scripts cued to linguistic and social conditions which we rely on 
with little conscious thought as we navigate the complexities of everyday life. When we are confronted with tasks 
w here the appropriate course of action is unclear, such as wTiting an important letter perhaps, or programming our 
VCR. or finding oui way to a location we have never visited before, we will usually be able to construct a relatively 
well strucUired problem space, and then find an appropriate path through that space with little difficult) . Studies of 
how^ experts in specialized areas of endeavour, such as physics, chemistiy, political science, medical diagnosis and 
more mundane activities such as taxi driving and t>ping, suggests that specialized expertise rests on the same 
cognitive foundations as the more commonplace expertise acquired in the course of ever} day life. In much the 
same way that people become familiar w ith commonly encountered tasks and problems, experts in specialized 
domains become familiar with the more esoteric tasks they' must accomplish in doing their e\^eryday work. And in 
much the same way that people develop and rely on automated, proceduralized responses and scripts when reacting 
to commonly encountered situations in everyday life, experts in specialized domains de\^elop and employ similar 
routine responses to commonly encountered tasks and problems in their domains. As Simon (1993, p. 403) has 
expressed it, “any expert can recognize the symptoms, the cues, to the bulk of the situations that are encountered in 
his or her everyday experience." Having done so, he or she can often readily retrieve an appropriate automated 
response from memory or. if such a response is not available, draw on relevant knowledge to construct a 
reasonably well structured problem space within which to search for a solution. Confronted with the same 
situation, non-e>q)erts in the domain concerned typically fail to notice or misinterpret the relevant cues, or if they 
do recognize their significance, they are often unable to rely on an appropriate proceduralized response and have to 
fall back on constructing a problem space through w hich the>^ will search for a solution path using whatever 
relev ant know ledge and operators they can bring to bear. 

Expertise, then, is associated with the possession of substantial amounts of domain specific declarative 
and procedural know ledge, and the ability to accurately retrie\^e and apply this knowledge to specific tasks and 
associated problems. Simon (1993, p. 403 ) notes that expertise in any domain probably requires the acquisition 
and integration of at least 50,000 familiar, retrie\ able memory chunks. His comparison in this respect to the 
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100,000 or more words in the vocabular>' of an educated person helps illustrate the dimensions of what might be 
called everyday linguistic expertise. Still, knowing a lot of words will not b\' itself ensure that someone can 
navigate the problem space defined b>’ the need to wxite a sentence or a letter. As noted earlier, one must also 
master the elements of grammar and st> le necessar>' to operate on the problem space. Moreover, and more 
interestingly, one must also develop an accurate and efficient means of locating and retrieving apjpropriate words, 
phrases and combinatorial rules from memory. Working through all the words that one knows alphabetically in 
search of an appropriate term will clearly not do. So too with the specialized chunks of domain knowledge held by 
experts in their specialist areas. As with everyday vocabular> , the knowledge which is the raw material of domain 
expertise appears to be stored in memor> in interconnected wa>s which facilitate task relevant access and retrieval. 
The schema construct has emerged as a powerful way of modelling this process. As VanLehn (1989, p. 569) put it 
“it looks as though schemas are the key to understanding expertise.” 

Schema 

As defined b>' Gick (1986, p. 102) a schema^ is “a cluster of knowledge related to a problem t>pe. It 

contains information about the typical problem goal, constraints, and solution procedures useful for that type of 

problem (Gick & Holyoak, 1983).” The more detailed definition proposed b>' Taylor and Crocker (1981) in their 

extensive review drawls attention to important organizational properties of schemata: 

A schema can be thought of as a pyramidal structure, hierarchically organized with more abstract 
and general information at the top and categories of more specific information nested within the 
general categories. The lowest level in the hierarchy consists of specific examples and instances 
of the schema (e.g. specific people or events). The schema is coimected to other schemas through 
a rich w eb of associations, particularly at the low er lev els of greater specificity, (p. 92) 

Experts’ domain specific schemata are thought to be much more richly organized and interconnected than those of 

non-experts. Thev' are considered to contain man> embedded sub-schemata, sets of pre-formed problem spaces with 

known solution paths and short cuts, many automated responses and scripts keyed to diagnostic cues, as w ell more 

hierarchical levels of abstraction (Rumelhart & Norman. 1983). As expressed b>' Reimann and Chi (1989, p. 172) 

“experts have developed several layers of this hierarchy, whereas only the first level ... seems to be developed for 

‘ Both Taylor and Crocker and Schacter (1989, p. 692) credit Sir Frederick Bartlett with the first use of 
the term schema in the psvchological literature. As quoted by Schacter, Bartlett (1932. p. 201) originalh defined a 
schema as ”... an active organization of past reactions, or of past e>q)eriences, which must always be supposed to be 
operating in any well-adapted, organic response”. 
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the verv’ beginning no\*ices. As skill is developed, the hierarchy develops into a complete tree with many levels of 
embedding/ These higher levels of abstract representation enable experts to recognize general principles, cues and 
“second-order features’" (Chi, Feltovich, & Glaser, 1981) characteristic of particular t\pes of problems which 
typicalh escape novices. 

Schemata ser\ e to guide the way people perceive the world b>' directing attention toward or away from 
aspects of a task. As illustrated in the Lesgold et al. (1988) stud>^ of X-ray interpretation, experts in this domain 
literally saw X-rays differently from novices, noticing more elements and making more connections between them. 
Schemata also ser\ e to filter and match task rele\^ant information to appropriate elements or variables in 
pre-formed problem spaces. These variables, or slots, are associated with default values that are assumed to hold 
until replaced with specific data, or fillers, from the task to be completed (Mandler, 1979: VanLehn, 1989). When 
the appropriate slots are filled with task specific information, a standard solution procedure can often be activated 
by firing an automated response or retrie\ing an appropriate script. This represents the most straight-forward form 
of schema driven problem processing, called routine problem solving by VanLehn (1989, p. 545). But while it may 
be routine in the sense that it pro\ides experts with a usually reliable and relatively rapid means of responding to 
commonly encountered problems, the amount of work to be accomplished may still be substantial. Extracting the 
relevant information from a task environment and then fitting it, or instantiating it, into a schema may be 
demanding and time consuming. When dealing with more difficult problems, domain experts also seem to rely 
heavily on their schematic knowledge, the major difference between experts’ responses to routine (for them) and 
more difficult problems hinging on schema selection. In Larkin’s (1983) stud>* of expert physicists’ responses to 
difficult problems, some immediately activated an appropriate schema, but others initially applied one or more 
incorrect schemata, only to abandon these in favour of another choice when it became clear thev^ would not lead to 
an acceptable solution. Such behaviour is behind Simon’s (1993, p. 404-6) earlier noted observation that expertise 
is largely a matter of recognition and habituated response. 

Reimann and Chi (1989, p. 180) caution “We do not yet have enough data about how experts work on 
‘difficult’ problems, at least not in physics.” Research based knowledge addressing problem processing in less well 
structured knowledge domains, such as administration and similar social action domains is even scarcer, and it 
may yet turn out that the theoiy’ of expert problem processing developed in these paragraphs does not capture or 
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satisfactorily address the more open and inherently uncertain task environment within which administrators act. 

The Voss and Post (1988) inquir>' into expert and no^’ice differences in responding to a political science problem 
constitutes one major, frequently cited stud>’ that addressed problem processing under uncertain conditions. Voss 
and Post’s subjects were asked to work on the design problem of increasing crop productivity in the then Soviet 
Union, The experts characteristically relied on two general tactics: decomposition, in which the main problem was 
broken down into a set of sub-problems, usually no more than three, and conversion, in which a sub-problem was 
converted into a solvable problem — that is one to which an existing schema could be applied. The initial diffused 
and ill-structured design task was thus broken down into sub-problems which could be addressed b>’ drawing on 
schematically indexed knowledge to derive an initial plan in much the same way as experts in physics and 
similarly well organized knowledge rich domains also responded to difficult problems. A important addition is that 
Voss and Post’s experts “built a case for why the proposed solution would work” (p. 274). In other words, in 
addition to devising a plan w hich satisfied their own criteria, the>^ recognized that any acceptable solution would 
have to satisfy fellow members of the relevant problem solving community. This is a condition which must be 
presumed to apply in other specialized domains where practitioners often encountered design problems, such as 
administration. 

Implications for study and practice in school administration 

The ingredients of the theory of human problem processing sketched above appear to offer a valuable 

framework within which to stud>^ the nature and practice of administration in general and educational 

administration in particular. Among the more ob\ious lines of inquiry presented are the identification of the tasks 

characteristic of particular positions, such as the school principalship^ exploration of how the degree of structure 

associated with or attributed to those tasks varies with experience, expertise, context, and personal characteristics 

such as ^•alues, training, leadership style and so forth; investigation of how experts and novices understand more 

and less highly structured problems in the domain— the nature of the problem spaces the>' construct; studies of how 

more and less expert administrators process problems — how they find their way across characteristic problem 

spaces; and inquir>^ into how' proficient administrators organize and access schematic memory— the cues they rely 

on to identity’ and classity* particular situations so as to retrieve potentially appropriate schemata together with the 
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procedures, scripts and other stored know ledge accessed through those schemata, and the slots and default values 
embedded in them. The implications for the improvement of professional practice and the training and selection of 
future administrators may be substantial. Building informed imderstandings of how more expert school 
administrators recognize, classilS’ and process important domain problems and claril>'ing differences between how 
they and less proficient practitioners imderstand the domain of practice will certainly provide useful models for 
critical examination and reflection in training and de\^elopmental activities. But while such models will 
undoubtedly be valuable, the theor>^ sketched above implies that the>’ will not provide shortcuts to the acquisition of 
expertise, which requires the mediated learning that can only be gained through direct experience. Indeed, the 
theory predicts that true novices will likely be unable to appreciate or e\ en fully imderstand the subtle and rich 
complexities of expert problem recognition and processing, and thus may be unable to learn much of value by 
directly studying ex'pert behaviour. More direct training benefits will likely stem from de^^eloping informed 
understandings of the tasks and situations novices see as unstructured and difficult, in the creation of appropriately 
sequenced learning and development activities for those preparing for administrative positions, and in the 
provision of supporting processes which may help new appointees learn more efficiently from their experiences.. 
Better informed models of administrative expertise will also help in the more reliable selection of mentors. These 
and other potential benefits will nonetheless require considerable direct research into the nature of administrative 
work and problem processing before the>' are likely to be sensibly realized. For this reason w^e share Leithwood and 
Steinbach’s (1995, p.314) concern that in the absence of a more solid research foundation, the rapidly growing 
popularit>’ of Problem Based Learning techniques in training school administrators (e g. Boud, 1985; Bridges, 

1992) may put the practical cart before the theoretical horse. 

Related research 

Four bodies of research literature are pertinent to the de\'elopment of problem processing approaches in 
educational administration. The first, ob\iously, is the cognitive science literature underlying the theoretical frame 
sketched above, particularly the past two decades of research investigating the nature of domain expertise. Reviews 
b> Ericsson and Chamess (1993), Reimann and Chi (1989) and VanLehn (1989) summarize much of this literature 
while Simon (1993) has recently provided a richly informed yet colloquially rendered account. The second bod>* of 
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pertinent research is the varied set of studies in the broader literature considering how administrators think. Some 
of this literature, such the studies of problem classification b>’ Cowan (e g. 1986 & 1991) and Day and Lord (1992) 
and the discussion of strategic decision making b> Schwenk (1988) is explicitly grounded in cognitive science 
theor>'. Other valuable contributions, including chapters in Srivastva’s (1983) The executive mind, and Isenberg’s 
(1984) How senior managers think adopt more inductive approaches. Earlier work, especially Mintzberg’s (1973) 
now classic The nature of managerial work which incorporated an information processing approach and some of 
Vickers’ (e g. 1984) essays make particularly insightful contributions to this literature, many of which are readily 
accommodated b> the cognitive theory of problem processing outlined above. 

The third bod>’ of literature consists of previous empirical and interpretive studies of educational 
administrators and their work which, while not conducted within a cognitive science framework, can yield data 
relevant to and interpretable within the problem processing approach. The Hemphill, Griffiths and Frederikson 
(1962) stud)’ with which we began provides a good case in point. One of their more pow erful and consistent 
correlates of principal performance as gauged b>’ both subjective ratings (p. 242) and scored responses to simulated 
tasks (p. 152) was participants’ scores on a standardized test of declarative knowledge about then current 
principles of school supervision and administration. While Hemphill et aL concluded from this that formal tests of 
professional knowledge could be useful in screening candidates for the principalship (p. 337), they' were imable to 
provide any coherent theoretical explanation for the predictive power of the observed relationship beyond the 
common-sense conclusion that more professional knowledge is a good thing. Contemporary' cognitive science 
theories of expertise predict and provide a coherent explanation for the observed relationship, possession and 
accurate retrieval of substantial domain relevant declarative knowledge from schematically organized memory* 
being engine of expertise. This should not surprise us, of course, as we should expoci newer and hopefully better 
theories to explain historical data while also predicting new relationships. Hence findings from previous studies, as 
in the example given. w*ill be pertinent to the future development of problem processing theory* and research. A less 
obvious bonus of cognitive theories of administrative problem processing is that in addition to accommodating 
empirical data they are also readily* applicable to extant interpretive accounts of school administration. Wolcott’s 
(1973) ethnographic stud)* of the work of principal Ed. Bell provides examples and illustrations of domain tasks 

and actions readily accommodated by* problem processing theory , as do Gronn’s (1984; 1985) studies and, 
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although the>- explicitly reject a cognitive approach in their introduction, Lighthall and Allen’s (1989) rich account 
of attempts at school change. The fourth bod>’ of literature is the research which has adopted an explicit cognitive 
science approach to the stud>^ of educational administration. 

Cognitive based studies of educational administration 

By far the bulk of this w ork has been conducted by Kenneth Leithwood and his associates, who must be 

credited with pioneering cognitive science derived research into school and school s> stem administration. 

Leithwood’ s initial interest in this line of inquiry grew out of his earlier studies of school principals which 

culminated in The principal profile (Leithwood & Montgomery, 1986). That work produced a behaviourally based 

stage model of principal effectiveness, in which the highest lev'el, most effective performers were described as 

“systematic problem solvers.” The theoretical framework accompanying the profile (Chapter 7, pp. 1 14 — 133) was 

derived from an information processing view of administrative action grounded in Newell and Simon’s (1972) 

early work on human problem sohing which was a precursor for contemporary cognitive science theories of 

problem processing. A subsequent study' by Leithwood and Stager (1986; 1989) explicitly incorporated major 

dements of cognitive theory' and provided the foimdation for fiuther work by the Leithwood group. 

The Leithwood and Stager (1986; 1989) study probed how 22 elementary principals, six of whom were 

designated as experts, attributed structure to six case problems and then thought their way through those they 

considered most and least structured. In general conformity' with the theory outlined earlier, analyses of the 

transcripts of how these principals responded to the case problems showed that while they were all able to retrieve 

automated responses from memory' for those they’ saw as well structured, differences were apparent between how 

members of the designated expert group and the more typical principals •viewed and thought about the presented 

problems, especially with regard to those they saw as less structured. While less structured situations were regarded 

as complex and difficult to solve by most subjects, members of the expert group typically regarded them as 

manageable and often outlined detailed plans— pro%'isional paths through the problem space— for dealing w’ith 

them. In contrast, principals in the non-expert group appeared less confident when dealing with the problems they 

saw as poorly structured, gave less attention to planning a response, and the plans they did develop were much less 

detailed and complete. In short, the designated experts seemed markedly more w illing and able to handle the 
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situations they saw as less structured than did the more average principals. Leithwood and Stager sxunmarized 
their findings b\' observing ‘‘our data suggest that in response to ill-structured problems, highly effective principals 
use processes similar to those used b> experts in other fields. We collected no obviously disconfirming data” 
(Leithwood & Steinbach, 1995, p. 66). Given that the bulk of previous research into cognitive models of expertise 
had dealt with more highly organized knowledge domains such as physics, chemistiy and chess, this was an 
important finding which cleared the way for further work in the less well organized knowledge domain of 
educational administration. 

Much of the subsequent w ork b\ the Leithwood group, which has included studies of individual and group 
problem processing b>^ principals and superintendents as well as an attempt at mapping the task domains of 
principals, has recently appeared in a collected edition of the original research reports (Leithwood & Steinbach, 
1995). Throughout this work the framework originally developed for analyzing the transcripts generated bv^ the 
original Leithwood and Stager stud}’ has been further developed and refined to yield what can be called the 
Leithwood model of administrative problem processing. This model contains three main processes; Understanding, 
which incorporates problem interpretation andgoa/ setting activities; Solving, which subsiunes identification of 
constraints to reaching a solution and solution processes employed in dealing with a problem situation, and 
Disposition, which refers to the values underlying and guiding a search for solution and the mood states of solvers 
as they work on problems. Within this framework, the more expert principals studied b>^ the Leithwood group have 
been found to typically (1) take greater care and work harder when seeking to understand difficult problems (seek 
to create more representative problem spaces) and anticipate that other participants will likely understand the 
situation differently; (2) seek to achiev^e multiple, generally broad, coherent and clearly articulated goals (develop a 
clearly defined goal state) while remaining open to new^ information and ideas; (3) anticipate 
constraints — difficulties to be overcome — and develop or retrieve from memory plans and procedures for dealing 
with or circumventing anticipated constraints (plan flexible paths through the problem space); (4) appear able to 
draw on a richer and more complete sets of solution processes (deploy generally appropriate operators for 
traversing the problem space); (5) are more aware of and more frequently articulate their values, which generally 
encompass commitment to the pursuit of socially approved educational and professional goals and practices 
together with respect for others; (6) t>pically appear calm and confident when working on problems. 
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There are currently ver>’ few other published studies that directly apply a cognitive based problem 
processing approach to educational administration. The collection of relevant work recently published b>* 
Hallinger, Leithwood and Murphy (1993) contains valuable chapters re\iewing pertinent theor>' (Ohde & Murphy: 
Wagner: Yeko\ich) and reports and studies of instructional applications (Hart: Kelse> : Prestine), but the only 
chapters reporting original research into how educational administrators worked on problems were two studies 
from the Leithwood group (Raun & Leithwood: Leithwood & Steinbach) and a report on some of our own work 
(Allison & Allison). A recently reported stud>^ b> Bullock, James and Jamieson (1995) helps shed light on how 
novice and expert educational managers in the United Kingdom view' and interpret their work, but neither the 
design nor the analysis takes explicit stock of cognitive theory as outlined above. 

The CASL Project 

Our Cognitive Approaches to School Leadership project had its roots in earlier studies b>' Nag> and P. 
Allison (1988. 1989) and Nag>^ and Moorehead (1990) of how principals thought through case study problems. 
Initial attempts at producing diagrammatic models of how subjects constructed and worked through problem 
spaces were unsatisfactory', but other lines of inquiry suggested that the use of checklists and similar kinds of case 
grounded marking schemes could identify meaningful differences in how principals worked on the presented 
problems. Building on this earlier work, we embarked on what we have come to think of as the first round study in 
the Principal Problem Processing strand in the CASL Project. Other studies underway as part of the CASL Project 
include inquiry into how principals view and talk about their work and problems (Allison, Morfitt & Allison, 

1996) and a study' of task complexity' and time-span of discretion in school systems based on Jaques’ (1976, 1989) 
theories of organizational depth structure and cognitive power (Allison & Morfitt, 1996), some elements of which 
were also addressed in our second PPP study' as discussed in the paper on cognitive complexity prepared for this 
symposium. 

As described by* Allison and Nagy* (1991), our first round PPP study collected data on how 29 elementary’ 
principals and 10 true no\'ice subjects thought aloud w'hen responding to a case problem in school leadership. As 
reported in Nagy' (1991) and Allison and Allison (1993) we made some progress de\ ising techniques for analyzing 
the think aloud transcripts generated in that study*. A second round of data collection involving 30 principals and 
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24 naive subjects was undertaken in 1993 and 1994 incorporating design features to overcome what we had come 
to recognize as weakness in the first stud> . This second stud>^ constitutes the prime focus of the papers prepared for 
this s>'mposium although some use is also made of data from the first round stud>'. The design of the second stud> 
together with an explanation of how \ arious measures of imputed expertise were derived is dealt with in one of the 
accompanying papers, others reporting results from different lines of analysis. This introductory paper moves now 
to consider two areas of continuing concern in studies attempting to explore how people represent and work on 
domain problems through comparative studies of experts and novices in the domain concerned, namely think-aloud 
protocols and the difficult problem of selecting appropriate expert and novice subjects. 

Think aloud protocols 

As Simon and Kaplan ( 1 989, p. 2 1 ) obser\ ed, ‘Protocol analysis, the use of verbal reports as data, has 
become an increasingly important (but often misunderstood) technique for studying human intellegence” — ^b\' 
which they mean the broad set of problem processing and design activities referred to in their earlier quoted 
description. Much of the misimderstanding associated with this method stems from confusion over purpose. The 
objective is not, as is sometimes assumed, to have subjects explain their thought processes as they process a 
task — to provide us, as it were, with their theor>^ of how they think. To the contrary, the objective of the exercise is 
to collect data, data which will then be anal>^zed to test and develop externally generated theories and models. 

These data may be generated in se\ eral ways, but the most widely used technique involves asking subjects to think 
aloud as the>' work on a presented problem. This technique generates what Simon and Kaplan call concurrent 
protocols. What the>' refer to as retrospective protocols are generated b>’ asking subjects to report everything the>' 
can remember about a problem processing activity immediately or as soon as possible after completion. We used 
both techniques in our PPP studies, ha\ing subjects think aloud as they worked on a case problem and then, when 
the>' had finished asking them to tell us eveiything they could remember about how they tackled the problem. 

Both episodes were audio taped and then transcribed. The conciurent protocols are anal>^ed to identif>' what 
elements of the case and related knowledge subjects heeded, and to tiy and map their thought processes. The 
retrospective protocols are used primarily to model how the case was understood and the goals subjects were 
seeking to achieve. 
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Another potential misunderstanding confuses data with action. The objective of a think aloud session, 
once again, is to generate data about how' a participant thinks about a problem, and not to predict how' he or she 

would handle a similar situation in real life. We would naturally expect some correlation to exist between the two 
circiunstances. for the theory under investigation holds that the subject will draw' on the same problem relev ant 
schematic memoiy in both cases. But this does not mean that a case response will or could model, still yet predict, 
how a subject might respond to a similar situation in reality. Regardless of how' realistic a presented case may 
appear, thinking about how' one might respond to a baldly described situation under clinical conditions is a vastly 
different proposition from having to actually deal with a similar problem w ithin a complex, contingent, 
information rich reality where the stakes are also real. Thus, faced with a real life, real time problem a principal 
might well adopt a plan of action ver>' different from the one she constructed when asked to think aloud about a 
case study describing a similar problem. The two response conditions are fundamentally different. Not that it 
matters. The main objective remains the generation of problem processing data and the bland case studv stands as 
a serviceable trigger to achieve this end. Moreover, while it would be desirable for some purposes to collect real 
time data about how' principals understand, think about, plan and deal with the real problems they encounter in the 
real world, this is currently and may always be impossible. We can, of course, collect data from and about real 
administrators in real contexts, and the accomplishments of the Leithwood group and others, particularly Gronn, in 
attempting this are noteworthy. Still, as every graduate student knows who has pondered trees falling in forests, 
the intrusion and presence of data gathering methods ensures the circxunstances will not fully equate with real 
reality. More to the point, given the limitations of current and likely future methodologv' and technology, attempts 
to probe how an actor understands and thinks about a real life situation in real time must necessarily rely on 
retrospective protocols and accept the limitations thev* impose. As the contributions of the Leithwood group show , 
much can be accomplished within these limitations. Even so, concurrent protocols generated from think aloud 
problem working sessions remain as the most powerful tool for generating the data needed for refining and 
extending our understandings of administrative problem processing. 
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The expert (and novice) problem 

Novice-expert studies employing verbal protocols have been the engine driving the development of current 
cognitive science theories of problem processing. Attempts at testing or applying these theories to administration 
in general and educational administration in particular face the difficult issue of reliably identifying experts and 
novices. Most of the pioneering studies dealt with domains with relatively well organized knowledge bases, such as 
physics and chemistrv . In such cases the novice expert continuum can be reasonably assumed to correlate with the 
time invested in learning and applying domain knowledge. Application, that is working with the domain 
knowledge b\^ frequently applying it to the completion of domain tasks, is considered to be crucial for the 
development of the more richly interconnected and extensive schematic knowledge thought to underlie expertise. 
On this basis VanLehn (1989, p. 560) explains that in much cognitive science research “The term ‘expert’ is 
usually reserv^ed for subjects with several thousand hours of experience” in the domain under study. By placing 
quotation marks around expert, he is alerting us to a less than obvious distinction between the “expert” subjects in 
most studies and those accorded similar status in others. This distinction lies at the heart of Hayes’s (1981) claim 
that it takes ten times as long, that is 20,000 hours or at least ten years of work and practice, for anyone to dev^elop 
“world-class” expertise. Clearly “world-class” experts are in a different categorv* from those qualifying under the 
initial standard recognized b>^ VanLehn, who might be more appropriately termed “everyday” e?q5erts. Some 
appreciation of the magnitude of the difference is given in a recent discussion of expert performance by Ericsson 
and Chamess (1994, p. 73 1) wherein they suggest that a performance level “at least two standard deviations above 
the mean level for the population” characterizes expert performance. How well this metric captures the difference 
in performance between an “everyday,” first level e:>q 5 ert, such as a regular tenured professor of cognitive science 
or administrative studies, and a world class, Nobel prize expert in these domains, such as Herbert Simon, is open to 
debate, but would not seem to misrepresent the differences involved. Even so, ten or more years of “intense 
application,” (Simon, 1993, p. 407) in a knowledge domain does not, bv' itself, guarantee that someone will 
become a “world class” e?q)ert, such as a chess grandmaster or a nominee for a Nobel prize. Clearly there are other 
factors at work, factors that influence how well practitioners learn from extended experience. Yet while 
practitioners with ten or more years of intense application in their domain may not display the outstanding skills 
and accomplishments associated with w'orld-class expertise, they could be reasonably expected to have developed 



CASE Project Paper #AERA96 1.2, revised July 1996. 



Problem processing & the principalship 
P 22 

greater expertise than those that meet the threshold standard of only several thousand hours of experience, a 
standard that allows for the not imcommon research practice of designating instructors of \mdergrad\iate classes as 
“experts” and their students as “novices.” 

As it turns out then, expert status is a much more ambiguous and variable categor> than it may at first 
appear. Indeed, it appears that “expertness” is best thought of as a continuous rather than a discrete variable, some 
experts being more expert than others. Expressed another way, the base variable of expertise is properly 
conceptualized as varying within as well as between novice and expert categories. Even within more highly 
structured knowledge domains, then, partitioning the expertise continuum to create a valid expert group may be 
problematic. VanLehn (1989, p. 565) also alerts us to a similar difficulty at the lower end of the expertise 
continuum, noting that some studies define the novice categor> in terms of subjects with no or ver>' little domain 
knowledge — which he defines as a “pre-no\ice ' category — whereas others recruit novice subjects on the basis of 
“se\'eral hundred hours of training (approximately a college course’s worth).” 

Becoming aware of some of the less obvious difficulties in identiiying novice and expert categories in 
more structured knowledge domains assists in grappling with the problem of partitioning the expertise continuum 
in the less well understood and less organized knowledge domains of administrative practice. One point that 
emerges is that defining a valid novice categor>' may be a more crucial design problem than might first be 
assumed. When we began our first rovmd PPP study we assumed that experience-in-role could ser\^e as proxy for 
administrative expertise, domain experience being the criterion imderlying most other novice-expert designs, as 
discussed above. On this basis we recruited subjects in four experience groups: Aspirants, individuals who had 
recently completed officially specified formal training but had not been appointed to the principalship; Rookies, 
principals with less than three years on-the-job experience: Seasoned principals, with between 10 and 15 years of 
experience, and: Veterans, w ith more than 20 years experience in role. We reasoned that combining the Aspirant 
and Rookies would create an appropriate novice group, and combining the more experienced groups would create 
an appropriate expert group while comparisons across all four groups would allow' for finer analysis. In accord 
with the previously made distinction between world-class and “ev^eryday” experts, we recognized that not all of the 
more experienced subjects would necessarily have learned as well or as much from their interaction with domain 
problems, and thus expected some to demonstrate greater proficienc>^ than others. This was indeed the case. What 
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we did not anticipate was that a few members of the novice groups gave better quality responses to the case 
problem we asked them all to process than did some members of the “expert” group. Closer examination of the 
background experiences of our subjects pointed to some plausible explanations for this. Whereas members of the 
expert group had an average of 19 years experience as principals as opposed to a mean of 1.6 years for the novice 
group, members of the expert group were on average only six years older than the “novices.” Moreover, there was 
only 8 years difference between the mean years of total full-time, in-school work exTierience in all possible roles. 
One of the important reasons for this two year difference was that members of the novice group had to meet higher 
qualification standards to enter teaching than did members of the Veteran category. Thus, while there were 
substantial differences in experience as a principal, members of the no\ice group had received more extensive 
pre-professional training than some members of the expert group, gained extensive if indirect exposure to 
administrative practices and problems through their in-school experience, and been actively' preparing themselves 
for entry to the principalship. All in all, then, the extensive domain relevant experience and training acquired by- 
the members of our initial Rookie and Aspirant groups clearly disqualified them as novices under the 
“approximately a college course’s worth” of domain training identified by VanLehn, let alone the zero experience 
pre-nov'ice standard. 

Our first round PPP study' thus gave us a salutary lesson in the complexities inhering in domain relevant 
experience in administrative work. Given the many opportunities for anticipatory socialization to aspects of 
administrative work usually present in formal organizations, together with the in-house training and other 
de^'elopment programs available for those aspiring to administrative responsibilities, then recently appointed 
administrators cannot be uncritically accepted as true domain novices. As well as satisfying applicable formal 
training requirements, some new' administrative appointees must also be expected to have gained variable amounts 
of potentially rele^'ant domain experience, through, for example, committee work, seix'ing in adrmnistrative roles 
in volunteer organizations, self-study , and informal mentorship. Selection and appointment processes typically 
recognize the potential rele\ ance of such experiences, and designers of novice-expert studies of adrmnistrators 
could profitably take note of such experience when recruiting potential nov'ice subjects. The professional culture of 
schools together with their ubiquity' in contemporary' society' further complicates matters when it comes to studying 
the school principalship. Teachers are typically aware of at least some of the problems with which principals must 
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deal: indeed they are often part of those problems and have a stake in how they may be resolved. Consequently it 

is not unusual for a principal’s actions or inactions on current problems to be critically re\iewed in staffrooms and 

elsewhere. Such discussions may often be ill-informed, but they nonetheless provide opportunities for 

non-principals to gain some domain relevant knowledge, although the knowledge generated fails to satisfy the 

important “intense application’' standard. So to with parents and other members of a school community who 

become involved in school problems in one way or another. While their exposure to the work of school leaders may 

often be e\^en more one sided than that of teachers, it will provide some exposure to domain relevant knowledge. 

Then again, it seems that many taxpayers who have no direct connection with schools are nonetheless willing to 
claim some knowledge about how' the>’ could be made better. As a result, it may be impossible to find more than a 
few , if any, educated members of modem society that would qualify as complete novices in some aspects of the 
operational, administrative, and leadership challenges facing school principals. The domain knowledge held b>' 
such external obser\'ers will necessarily be partial and fall far short of the rich, schematically organized knowledge 
expected of expert principals. But that is not the point, for it may qualify them as novices under the criteria 
specified by VanLehn. 

We attempted to control for these confounding factors in our second PPP study when recruiting members 
of our novice group. We sought to approximate the standards discussed by’ VanLehn b>' recruiting an initial set of 
novice subjects from student teachers — who have had some professional training and school experience, and thus 
approximate VanLehn’ s working definition of novices — and an additional set of graduate students from domains 
other than education, who approximate VanLehn’s true naive pre-novices. The initial comparison groups in our 
second PPP study' were thus the novice group as described above and a group of practising principals. For the 
purposes of this initial comparison, all of the principals were recognized as experts solely by virtue of being in a 
school leadership position. As hopefully demonstrated in the preceding paragraphs, this design conforms to the 
principles underlying most other novice-expert comparisons in other domains. As such, we reasoned that it would 
provide a more robust and potentially fertile framework within which to test the presence of novice-expert 
differences identified in previous studies, as well as to isolate elements of schematic knowledge developed b>' 
school principals, all of w hich can claim at least first level, “everyday” expert status, solely b>' virtue of their 
position. 
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The problem of differentiating lev els of expert performance within the principal group remained. That 
there are variations in the level of performance among principals cannot be denied. Indeed, it may not be 
unreasonable to expect that the performance of some may approach or even match the two standard deviations 
above the population criterion suggested b> Ericsson and Chamess, where the population would be all principals 
appointed under a set of broadly comparable criteria as manifest at a national or culture area level. Under the 
Ericsson and Charness metric, such “national class” principals would clearly be statistical outliers, and thus verv’ 
few in number. It follows that truly exceptional performers would likely not be captured in a random or opportumty 
sample of the population but would have to be deliberately searched for. Given that there is no standard and 
reliable comparative method for ranking principal performance as there is, for example, in chess and contract 
bridge, then any attempt at identifying exceptional principals (and other administrators) will have to make use of 
other criteria, especially reputational indicators. The obvious problem with reputational data is that it may be 
inaccurate. Because of this Ericsson and Charness caution that rather than relying solely on reputation when 
searching for outstanding experts, researchers (and others) should also look for evidence of superior performance 
(p. 732). This is the approach adopted b>' the Leithwood group. In the initial Leithwood and Stager (1989) stud>' 
and some others, members of their designated expert group had to pass through two screens. First, two central 
office administrators were independently asked to nominate highly effective principals. Second, the nominees were 
interviewed to assess their self-reported administrative behaviours against the criteria associated with high 
performance contained the The Principal Profile (Leithwood & Montgomery, 1986). Of the 22 subjects in that 
stud) , six — a little over a quarter — satisfied both the reputational and the more objective tests of superior 
performance. Even so, this does not necessarily mean that those designated as experts by these methods satisfied 
the statistical outlier criteria suggested b>^ Ericsson and Charness. 

Still, the Ericsson and Charness metric might not apply to domains such as administration where 
practitioners are required to act under conditions of uncertainty. As discussed by Johnson (1988), a number of 
studies of ev erv day experts in diverse domains such as graduate admissions, economic forecasting and medical 
diagnosis have documented what appear to be disappointing lev^els of performance. In a stud)' of bankruptcy 
judgements, for example, an unw ieghted, single variable regression model predicted business failures much better 
than did a sample of experts in that domain (Johnson, 1988, p. 212, Libb)', 1976). Johnson (1988. p. 212) 
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comments, that in these studies of problem processing in social action domains ‘‘The superioriu of [everyday] 
experts to novices is surprisingly small.’' Yet when one reflects on the performance standards that might ser\ e to 
distinguish better from more average practitioners in uncertain task environments such as administration, it seems 
unrealistic to expect even the most proficient performers to be universally successful in dealing with the wide 
variety of situations with which they are confronted. Indeed, it is surely unreasonable to expect that expert 
principals could create and consistently maintain, in the words of my colleague Jim Sanders (1983, p 12), schools 
with “unfailing leadership, error-free decision-making, unsinkable faculty morale.” If we accept this, then what 
might constitute standards of exceptional performance in the principalship? Leithwood (Hallinger, Leithwood & 
Murphy, 1993, p. 271: Leithwood & Steinbach, 1995, p. 13) has contributed some valuable obser\^ations on this 
point. He begins b>* noting that the extensive amounts of domain relevant know ledge and skills held b>' experts 
does not guarantee success in achieving desired goals in difficult circumstances. What counts, he suggests, is an 
individual’s accumulated record of accomplishment, rather than success in achie\ing a given goal, completing a 
particular task, or handling a specific problem. He then notes that the standards conventionally applied to judge 
what constitutes an acceptable or exceptional record of performance vary enormously across domains, depending, it 
appears, on the degree of control which indi\iduals have over the task environment in which they' must act. Thus, a 
designated hitter in professional baseball is considered to have established an exceptional record of performance by 
hitting only 30 per cent or so of the pitches faced during his career. Might a similar standard of accomplishment 
hold for how' well principals deal with the problems encountered in their domain? I suspect that we would want to 
apply a higher standard, but in seeking to do so we would have to take school and environmental variations into 
account. We would expect an exceptional principal to have a lower batting average — record of performance — in a 
difficult school where she is able to gain less control over the task emironment than she would have in a school 
where the internal and environmental circumstances are more conducive to success. In the latter circumstances it 
would be reasonable to expect a markedly higher batting average, but it would surely be impossible for even the 
most outstanding principal in the most conducive environment to “bat a thousand.” 

A necessary' consequence of the preceding analysis is that some principals will have poorer records of 
performance than others and, if we accept that those with outstanding performance records will fall short of 
perfection due to the inherent complexities and uncertainty of the w'ork to be done, then we may also expect that 
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poorer principals will have depressingly low “batting averages.” This produces the semantic paradox of the poor 
expert. It is a paradox which nonetheless helps drive home the broad point developed above — that some experts (as 
defined in terms of extended exposure to domain problems) will have better, and some worse, records of 
performance than others, and that the variation may be substantial. The important question that follows is why 
should this be? Within the theoretical framework adopted for the CASL Project as reviewed in this paper, two 
main explanations emerge, which are in effect two sides of the same coin. The first is that while less proficient 
principals were able to acquire the declarative and procedural knowledge (or at least sunive the formative and 
training experiences deemed necessary to acquire such knowledge) so as to satisfy initial appointment standards, 
the>' were somehow imable to organize and integrate this knowledge in schematic memoiy' so as to ensure the 
usually reliable and efficient retrieval necessar>^ for satisfactory^ performance. In such cases, poor job performance 
is directly attributable to weak training and selection procedures. The second explanation is that while more 
proficient principals were able to learn successfully from their on-the-job experiences, and thus enrich their domain 
schemata, less proficient principals did this less well — they were unable to learn from their work experience as 
efficiently as others were. Ideally, this “learning curve” explanation for variable levels of proficiency should also be 
detectable through training and selection processes, but the cognitive and other factors underlying a person’s 
ability to learn rapidly and efficiently from a new and more challenging work emironment are not sufficiently well 
imderstood to produce reliable tests. It does appear that the cognitive elements involved are not well correlated 
with whatever is measured b>^ standard IQ tests. There are reasons to suspect, however, that various theories of 
practical intelligence, such as Wagner and Sternberg’s (1986, 1990) “street smarts” and related theories of 
cognitive power (Jaques, 1986) and cognitive complexity (Streufert, & Nogami, 1989; Streufert & Swazey, 1986) 
may apply . 

In designing our second PPP study^ we attempted to capitalize on many of the points made above. We 
recognized that some members of our first le\ el expert group, that is the principals participating in the study , 
would likely be more oxjpQTi than others. Given this, we wanted to find a reasonably valid and defensible way of 
distinguishing between both more and less proficient principals so as to facilitate comparison and analysis of the 
think-aloud transcripts. In line with the reasoning presented by' Ericsson and Chamess we recognized that we 
would have to make use of reputational data, but would also have to incorporate some independent screen, as was 
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done b>’ Leilhwood and Stager. Even so, we wanted to avoid having potential subjects nominated b>’ peers or 
subordinates so as to minimize halo effects during data collection. We eventually decided to rely on a multiple set 
of performance measures, all of which would be collected independently of the think-aloud session. Principal 
participants were recruited by simply requesting volxmteers from area school systems, who were invited to spend a 
day with us on campus, where they participated in the think-aloud session and other data collection activities. At 
the end of the day, we asked their permission for us to seek ev aluations of their work as a principal from three 
subordinates (teachers) and two superordinates whom they were invited to nominate. We then mailed a 
questionnaire containing various rating scales to these nominees with an appropriate explanatory letter. These 
ratings provided us with our reputational data. We also sought another measure of work performance by having the 
principals and some of their teachers complete the Index of Perceived Organizational Effectiveness (Hoy & Miskel, 

1991, p. 400). Once all these data collection activities had been completed, but before the data were anal>'zed, we 
convened a jur>^ of reputationally expert principals who rated the quality of responses given to the case problem 
used for the think-aloud session. A set of decision rules were then dev ised to identify , on the ev idence collected, 
those principals who appeared to have achieved a record of superior performance in the eyes of nominated 
co-workers, and who had also demonstrated superior performance in processing a sample task, as judged b>^ a jury 
of their peers. Of the 30 principals in our first level expert group (as opposed to the true novice group) six (20%) 
satisfied the relevant criteria to be assigned to the High imputed e?q)ertise category for the purposes of our 
analyses. A separate set of criteria w ere developed to identify potentially weaker principals on the basis of the 
reputational and school effectiveness data. Five of the 30 were subsequently classified as falling into a Low 
imputed expertise category*. 

In siun, our second round PPP study' which provided the data discussed in the other papers prepared for 
this symposium was designed so as to allow' true novice-expert comparisons as well as proficiency' based analyses 
within the first lev el expert group. To try and avoid the confusions associated b>' referring to different levels of 
expertise, the papers that follow w ill generally refer to the various comparison groups in terms of their members 
rather than their theoretical status, using the following conventions, the true novice group will generally be 
referred to as “others,” and the first level expert group as “principals”; within the true novice group, the novices 
will be referred to as either B.Ed. students or novices, and the pre-novices as naives: within the principal group. 
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those identified as more proficient will be referred to as members of the High imputed expertise group, principals 
identified as least proficient are classified in the Low imputed expertise category', with the remainding principals 
comprising the Medium imputed expertise group. 

Conclusion 

This paper has reviewed the theory and literature base for the remaining papers prepared for a 1996 
AERA sy'mposium reporting aspects of our C ASL Project. The second paper in the set reviews the research design 
adopted for our second PPP stud\’ in some detail. 
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